OOCOHEIT BESOHE 

EH Oil 706 

Judd, Hilson A.; And Others 

Defrelopaent and Foraative Evaluation of a Five flodule 
Coaputer-Hanaged Instructional System for Educational 
Psychology. Technical fieport Nuiber Nineteen. 
Texas Univ., Austin. Computer-Assisted Instruction 
Lab. 

National Science Foundation, Hashington, D.C. 

TB-19 

Apr 73 

85p. 

I!F-$0.65 HC-$3.29 

Academic Performance; Behavioral Objectives; 
^Computer Assisted Instruction; Course Evaluation; 
Criterion Referenced Tests; Curriculum Development; 
^Educational Psychology; Formative Evaluation; Highrr 
Education; Individualized Programs; Instructional 
Design; Instructional Materials; Program Costs; 
Teacher Education Curriculum 
'i'Computer Managed Instruction 



This report describes the design, development, and 
formative evaluation o£ a computer- managed instruction (CMI) system 
in a university setting. The system, implemented on an IBM 1500 
Instructional System, is designed to serve an undergraduate 
educational psychology course for teacher trainees. Tvo broad goals 
are defined for the project: satisfactory post- instructional student 
performance; and positive affective outcomes Mith regard to the 
subject matter and the instruction/evaluation system. Subject matter 
is structured into five modules: 1) Computers in Education; 2) 
Classroom Management; 3) Tests and Measurement; 4) cultural 
Differences; and 5) Statistics. The introduction, behavioral 
objectives and sample test items for each module are presented in 
appendices to the report. Only the data regarding performance 
measures are p:7esented in this report. Evaluatiqn of the affective 
aspects of the system will be discussed in subsequent reports. Based 
on project trie:ls and initial use, modules and evaluation instruments 
have been revis^^d. (JB) 
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INTRODUCTION 



Until fairly recently, major interest in the instructional use of 
computers has centered around various forms of computer-assisted instruction 
(CAI), a*il of which involve intensive interaction between the student and 
computer. Oue to the amount of effort required to specify the details of 
this interaction adequately, tne design and implementation of such programs 
is an expensive and time-consuming process. Consequently, most of the CAI 
programs currently available are fairly short and pertain to only a very 
small portion of a complete course, As a result, one often finds short seg- 
ments of CAI embedded in a course which is otherwise taught in a traditional, 
non-individualized way. Furthermo^^e , since they can bear only a small portion 
of the total instructional load, such short programs usually represent addi- 
tional expense rather than approaching justification as a replacement cost. 

An alternative approach ir to make use of the computer's information 
processing and memory capabilities to manage the individualization of a 
complete course or the major portion of a complete course. Under a computer- 
managed instruction (CMI) system, the instrur on itself is not presented by 
computer. Rather, the instruction is preser, d by means of conventional, less 
expensive media. While the most commonly use 1 medium is the printed page, 
slide presentations, audio tapes and films could easily be incorporated. The 
student proceeds to study at his own pace while the computer is used to monitor 
and, to some extent, direct his progress through the instructional materials. 
This is done by means of testing at frequent points in the program to diagnose 
the student's strengths and weaknesses, providing prescriptions for remedial 
work, and scheduluj students' use of the available instructional resources. 
Short segments of CAI might, of course, be among the resources available for 
assignment by the CMI system. While similar programs of individualized 
instruction have been conducted without recourse to a computer, the magnitude 
of the management activity required becomes overwhelming when large numbers 
of students are involved. 
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The purpose of this report is to describe the design, development, 
and formative evaluation of a CMI system in a university setting. The system 
was designed to serve an undergraduate educational psychology course for 
teacher trainees at The University of Texas at Austin. It was developed by 
The University of Texas Computer-Assisted Instruction Laboratory and imple- 
mented on an IBM 1500 Instructional System (IBM, 1967). The major source 
of funding for the project was the National Science Foundation. 

Two broad goals were defined for the complete CMI project: satis- 
factory post-instructional student performance, and positive affective out- 
comes with regard to the subject matter and to the in^truction/eva ' i ation 
system. Only the data regarding performance measures will be presented in 
this report. Evaluation of the affective aspects of the system as measured 
by expressed student curiosity about the subject matter, reported anxiety 
during testing, and responses to an evei'jation questionnaire, will be dis- 
cussed in subsequent reports. 

One major purpose for which the development of this system was 
funded was to provide a vehicle for educational research. While research has 
been and continues to be conducted within the context of the syst>im, the 
research itself will not 'jc discussed here but will be presented in detaM 
in reports pertaining the specific projects (e.g., Richardson, O'Neil, Grant, 
& Judd, 1973). The environment in which this research is tak.ng place dnd 
the factors which were instrumental in creating tfiis environment will be 
described here. ^ 



INSTRUCTIONAL DEVELOPMENT 



Tne student performance data to be reported were derived from 
student scores on criterion-referenced evaluation instruments desiuied to 
measure achievement of the ins'^.ruction' s objectives. Since scores on in- 
struments of this type do not lend themselves to analysis by the traditional 
statistical measures of validity and reliability, the meaning of these scores 
may- be ser'^" usly questioned unless there is a strong reason to believe that 
the procedures used for designing the instruction and developing the evdlua- 
tion instruments were adequate (e.g., Glaser, 1963; Livingston, 1972; Kriewall, 
1972; and Edmonston, Randall, & Oakland, 1972). For this reason, the charac- 
teristics of the product development group and the instructional design pro- 
cedures full owed will be discussed in detail. 
Product Development Group 

Tne instructional product development gr'oup consisted of the two 
project directors and seven graduate students, divided into two-person teams. 
Typically each team contained a content matter specialist and an instructional 
design specialist and each -ceam was given the primary responsibility for 
developing one module. Because of a shortage of qualified personnel, a few 
of Lhe students sei^ved simultaneously on two teams. The project directors 
provioed the management required for the coordination of the teams. This 
included monitoring the work of all teams to insure that a standard of quality 
was maintained, providing the expertise necessary to solve problems beyond 
the competencies of the students, supervising the allocation of resources such 
as secretarial time, and providing liaison with other fdcui^y members involved. 
Although the basic production unit was -'he two-man team, occasionally all 
teams met as a group to report progress and to have their work evaluated. 
However, most inter-team communication was mediated by the project directors. 
A schematic diagram of the management/communications network is presented in 
Fi gure 1 . 
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PD--Project Directors 
I--Initructiondl r ?sign Special ist 
C--Content Matter Specialist 
<^ --x..dicates strong interaction 
<r -> --Indicate", weak interaction 
*--The same person filled both 
roles on Team Four 




Figure 1 .--Management/communications network 
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Instructional Design Procedure 

A flowchart is presented in Figure 2 to summarize tne instructional 
design procedure used to develop the modules. Although all of the finished 
modules were similar in fonn, slightly different problems were encountered 
during the developnent of each module. To reflect the fact that the design 
procedure varied across modules, the flowchart is presented in a "branching" 
format where the branches indicate alternatives which were followed in response 
to specific problems. 

The first step in the instructional design process consisted of an 
assessment of student needs. The assessment was basically an attempt to 
answer two questions: What information and/or competencies that are valuable 
for secondary school teachers to possess are not being taught within the 
current curriculum? What information and/or competencies currently being 
taught to secondary school teacher candidates can be taught more effectively 
and efficiently using computer-managed instruction? (For a complete discussion 
of the identification of needs, see Gage, 1970). These questions were dis- 
cussed with the class instructors, with the faculty course coordinator and in 
meetings of the instructional design group. As a result of this needs assess- 
ment, five topics were selected: (1) the use of computers in education; (2) the 
use of behavior modification -".jchniques for classroom management; (3) the rela- 
tionships between cultural differences and achievement in the school environ- 
ment; (4) the use of teacher-made and standardized tests and principles of 
measurement; and (5) the use of statistical tools to interpret test scores. A 
constraint placed on the design was that the modules require no more than an 
average of four hours apiece in student study time. 

After the topics were selected, the final two graduate students 
acting as content matter experts were selected and students were divided into 
the five instructional design teams. This division was made on the basis 
of competencies and interests. An instructional designer was paired with a 
content matter specialist whenever possible, thus allowing for the most effec- 
tive use of the available talent. Content matter specialists were not required 
to spend time learning instructional design skills, and instructional designers 
were not required to spend time learning content matter. Thus, the skills 
which each individual brought to tho project were immediately usable without 
the need for extensive training. 
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Figure 2. —Instructional design system flowchart. 
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Each team was given the responsibility for analyzing iheir specific 
^jpic and deterrnininq a set of behavioral objectives which could be achieved 
within a computer-managed instructional module. This specification of ob- 
jectives relied heavily on the judgment of the content matter specialist. 
The instructional designer assisted the content specialist in stating the 
objectives in behavioral terms. The designer also classified each objective 
as being either a knowledge, comprehension, application, analysis, synthesis, 
or evaluation objective in order to be sure that certain categories were not 
being overemphasized. The criteria presented by Bloom (1956) were used for 
this classification. 

Once the objectives had been defined, the teams again met as a 
grojp. Each team presented -md defes'Oed the objectives which it had formu- 
lated. Both the value of each objective and completeness of each set of 
objectives were considered. The objectives for each module were revised until 
the group agreed that they were satisfactory. 

During the next stage each team wrote three to six multiple choice 
test items to evaluate mastery for each objective in thei>" module. The 
teams then met again as a group to review and criticize each other's work. 

Following the specification of the objectives and evaluation tost 
items, the next task was to secure the instructional materials. Generally, 
the first step was to search for suitaole mate/^ials available at little or 
no cose. For example, the team which designed the measurement module selected 
some materials which were available at no cost from Educational Testing 
Service (1971). If available n;aterials were inadequate, the team designed 
and produced a dra't of the instructional material which they felt was necessary 
for the achievement of the stated objectives. 

Afte ' materials had been secured or developed, the teams again met 
as a group to evaluate the materials selected. The critical question asked 
was whetfier the specified objectives could be achieved with these materials. 
If the answer was '*no*', a decision was made regarding whether to change 
materials or revise the objectives. Since these decisions had to be made on 
the basis of "expert" judgment, a gr"'»o decision was considered preferable 
to decisions hy individuals in order insure a higher degree of accuracy 

(e.g., Smith, Stanley, & Shores, ^.957). 
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Once the instructional materials had been selected, the validity of 
the evdluation items was reconsidered in order to avoid the po:.sibility 
that the materials were teaching only the test rather than the desired 
behavior. For example, in order to test the achievement of an objective 
requiring the application of a principle, the student must be required to 
apply the principle in a novel situation (Bloom, Hastings, & Madaus, 1971). 
If a test item requires the application of the principle to a familiar situa- 
tion, the item has no validity. Items judged to be invalid by the instructional 
aosigners were replaced. Since this judgment was considered to be rather 
straightTorward, each team established the content validity of their own items. 
The items were then divided into three sets. The other two sets were used 
to produce two equivalent forms for evaluating student performance following 
1 nstruction. 

Pilot Evaluation and Revision 

At this point, the modul iS wore ready for an initial, small-scale 
trial. Each of the four modules was administered to one or two classes of 
students (n = 30 per class) registered in the Educational Psychology course 
during the 1972 summer session. The fifth module. Cultural Differences, was 
not completed in time for a summer trial. During this initial trial, the 
evaluation instruments were administered via paper and pencil tests and each 
student was presented with all items from both form? of the test. 

Test results and student attitude data gathered by means of a 
questionnaire were used as a basis for revising objectives, instructional 
materials, and test items. The test scores obtained were somewhat higher 
than had been anticipated but this was considered to be at least partially 
an artifact of mode of test administration. The test items had been developed 
for computer administration, in which case the student would see only one 
item at a time and would not be able to skip an item and return to it later. 
This minimizes the possibility that information in one item can serve as a 
cue to the answer of another question. In contrast, students could skip 
itoms and return to them later in the paper and pencil form of the test. In 
addition, course instructors maintained that summer students appeared to 
be somewhat uirferent from stuaonuS taking the course during the regular term, 
and, thus, revisions were made with caution. 
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In general, about 20 percent of the instructional materials 
anti 30 percent of the test items were replaced or underwent some degree 
of revision. The format in wni^h the modules were presented was modified 
slightly and an introduction to tn*^ set of modules was written. Factors 
which seemed as though they might become problematic were identified. For 
example, while certain materials were only moderately effective, the data 
did not clearly indicate that these materials should be replaced or revised. 
Therefore, such materials were retained for further evaluation. 
Final Module Format 

Following revision, the instructional modules and an intro- 
duction to the modules were published "in house" as Educational Psychology 
Module s , 1972. The format of each module was similar and consisted of 
an introduction te themodule, specific reading assignments, each with 
a, short overview of the reading assigned, behavioral objectives ana sample 
test questions. 

The introduction was designed to function as an advanced organizer. 
The assigned readings, of course, contained the material which the student 
is expected to learn. For the most part, the readings assigned were contained 
within the modules themselves. However, the majority of the reading materials 
for the Classroom Management and Statistics modules were contained in text- 
books to be purchased by the student. A portion of the Tests and Measurement 
module was contained in materials supplied by the Educational Testing 
Service. The reading .materials for the Computers in Education module were 
supplemented by a short demonstration of computer-assisted instruction. This 
demonstration also contained instruction on the operation of the terminal 
and served to familiarize the student with the facilities and procedures of 
the Computer-Assisted Instruction Laboratory. 

The objectives specified, in behavioral terms, the competencies 
which the student was expected to exhibit following instruction. Sample 
test questions keyed to the objectives served to further define the objec- 
tives for the student and to familiarize him with the type of instrument 
by which he would be evaluated. 
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In order to provide a more complete description of the content 
, of the modules, the introduction, behavioral objectives; and sample test 
items for each module are presented in /Appendices A through E. The material 
is presented exactly as it appears in Educ ational Psychology Modules . 1972. 
The reader who is unfamiliar with modularized instruction is encouraged to 
examine these appendices before proceeding. 



TEST PROGRAM AND DATA MANAGEMENT SYSTEM 



Test Program Development 

The final form of the evaluation instruments was not determined 
until after analysis of the results of the summer session trial. Two 
parallel test forms were defined for each module » each containing one to 
four questions per objective. The primary motivation for the construction 
of parallel forms was to provide ? means by which each student could be tested 
twice over the same material^ either on a pretest-posttest basis on a 
posttest-retest basis. A secondary consideration was to increase test 
security. The availability of different forms allowed students to be 
randomly assignee to one of the two forms. As a further test security 
precaution, two fferent sequences of items were defined within each fom. 

The number of questions per objective was determined on the basis 
of the module authors' rating of the importance of each objective. While it 
would have been desirable to include at least four items per ojjective, total 
test length was constrained by the amount of time which students coiili be 
asked to devote to evaluaticn. With the exception of statistics, e^o) module 
test form contairied from 15 to 30 items. The statistics moduie contained 
a total of 60 items per foni, but the complete module test was split into 
five subtests corresponding to the five units within the module. The subtests 
n^ere administered to studerts one at a time and were treated as separate tests 
for programming purposes. All test questions were of a multiple choice format 
anc., with only two exceptlcns, each contained fotr alternatives. 

Kaster copies of the test questions and their correct alternatives 
were typed and kept on fil-?. Each test question was formatted on a planning 
guide representing the character spaces available on the IBK, ISOO system 
cathode-ray tube displays. Cards punched directly l>om these planning guides 
served as input to a prep ocessor program run on the CDC 6600 computer system. 
This program produced Covrsewrit^r II code for the IB?i 1500 system. (The 
preprocessor program is iescribcd in Mitchell & Conner, 1S71.) The code 
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produced by the preprocessor was then integrated into the control program 
written by the 1500 system programmers and assembled. Due to the program's 
relatively simple decision logic and the use of the preprocessor, only a 
limited amount of on-line debugging time was required. The test program 
control logic and the characteristics of the data management system are 
described in the following section of this report. 

In their programmed form, test items for a particular module were 
presented one at a time on a cathode-ray tube display. Students responded 
by typing a 1, ?, 3, or 4 corresponding to the response alternative selected 
A response other than one of these four characters elicited a message to the 
student requesting him to type only a 1, 2, 3, or 4. 

Students were given diagnostic feedback on their performance 
immediately following their completion of each test. This consisted of 
the student's total score on the test (as a percentage), the criterion per- 
centage set by his instructor, and his percentage score on each objective. 
A typewritten copy of the same feedback was also available if the 
student requested it. Two types of reports summari:zing student performance 
were available for the class instructors. These reports were also prepared 
on the typewriter terminals and were available to the instructors within a 
few minutes after being requested. The characteristics of these reports 
are described in more detail below. 
Data Management System 

The goals directing the design of the CMI data management system 
were to provide flexible, on-line testing for a large number of students and 
to facilitate on-line report generation. Since the conventional 1500 system 
response data storage and retrieval process involves the relatively cumber- 
some use of magnetic tape, the requirements of CMI dictated the use of disk 
for bookkeeping and studeni: response data storage. 

In the current version of the disk centered system, data for up 
to 640 students are stored on a disk pack and accessed for on-line summaries 
The file structure is designed to accommodate up to three class sections for 
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each of up to 20 instructors. Each class section may contain uf, to 60 
students, subject to the O'erall constraint of 640 students. The ii.^onnation 
contained on the CMI data management recording pack is of four types: ook- 
keeping information, course configuration information, section cof*f iguratlon, 
and student data. 

Bookkeeping information consists of pointers to available files, 
a current count of the instructors and sections retjistered on the data pack, 
and a current count of the number of students registered over all class 
sections. Tnis information is accessed during initialization ov the data 
pack, i.e.. Instructor, sec^'ion and stuJsnt registration, prior to use of 
the CMI tests by students. The structure in wnicn this information is stored 
is represented in Figure 1 of Appendix F. 

Course configuration information specifically defines the content 
of each CMI module to be administered. Two sectors on the disk pack (see 
Figure 2, Appendix F) are allocated for this information. The first word 
of the first sector indicates the total number of modules (up to 25) avail- 
able. Sequentially thereafter are a set of cp to 2b 26-word records which 
specify the number of objectives and the number of test items per objective 
for each module. A given module may consist of up to 20 objectives and each 
individual objective may have up to ten test items. Course configuration 
information is utilized in detenrnning performance (percefit correct) for a 
given objective within a module and in determining overall performance for 
the module* This information is also used in producing the suiTvnary print- 
outs for instructors and in providing performance feedback to students. The 
course configuration is specified on-line by a proctor through the use of a 
program written in Coursewriter II specifically for setting up of the data 
pack and for on-line retrieval and summary of student data. 

Section configuration information specified the module tests which 
are to be administered to students In a given class section, the sequence in 
which these tests are to be administered, and the performance criterion 
assigned for mastery for each module. Thus, each section can have a module 
assignment and sequence which is unique to that section together with a 
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unique set of performance criteria. This allows the use of a single instruc- 
tional program containing all irodules to be used for a number of class sections 
with differing requirements. The instructional program's control routine 
selects only the module tests specified {and selects them in the appropriate 
sequence) for administration to a student in a given section. Each class 
section is identified by a two character ident1fier--the first two characters 
of the student number. The structure in which this information is stored 
is outlined in Figure 3 of Appendix F. Section configuration information 
is used to determine whether or not a student has achieved the specified 
criterion both at the objective and module levels. This information is also 
used in producing summary printouts for the instructors and in giving perfor- 
mance feedback to students, 

Student data stored on the data pack consist of a set of eight- 
word records in the format shown in Figure 4 of Appendix F. Within a record, 
raw scores are maintained for each objective In a module together with latency 
data for the module test, and date (month and day) of test administration. 
During the formative evaluation, data from the curiosity ano anxiety scales 
administered with each module were also maintained with each record. An 
otherwise unused portion of the recording pack was used to store student names 
which could be accessed posi tional ly . One eight-word record is written to disk 
upon completion of a test or retest. Records for one initial test and up to 
two retests may be stored fc each module* The series of records written to 
disk for a given student are all contained within a two-sector block reserved 
for that student. Student oata records are utilized in producing surmary 
reports for the instructors. 

During the formative evaluation^ allocation of each student perfor- 
mance record block was handled during section registration and configuration. 
The maximum number of students anticipated for a given section had to be 
ent^?red by the proctor .at section registration time. From this information 
and certain bookkeeping information, an N-word table (where N Is the number 
of students) of pointers to recording blocks was set up for that c^ass section. 
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Thus, when a student in a given section initially signed onto the system, it 
was only necessary to determine his section (from the first two characters 
of his student number) ana to find the pointer to the allocated recording 
block by indexing. the table of pointers using the binary value of the last 
two characters (digits 01-60) of his student number. 

The requirement that tne number of students in each section b^'^ 
entered prior to the time the stuaents signed onto the system was found to 
be unsatisfactory during the formative evaluation due to xhe number of students 
adding and dropping the course after registration. The scheme has since been 
modified to dyrimically allocate blocks at the time of initial student sign 
on. 

As was mentioned above, one of the primary purposes of the CMI data 
management system was to facilitate the generation of timely and meaningful 
reports to instructors. This is done by use of a ''course" coded in Coursewriter 
II which performs report generation as one of its functions. Reports to in- 
structors may be generated at two levels: a course summary report of informa- 
tion at the module level and a module summary report of information at the 
objective level . 

A course summary report is specific to a given class section and a 
given instructor. The report is headed by the instructor's name and the 
section number, followeJ by a listing of all modules to be tak'^n by students 
in that section in the sequence specified in the section configuration record* 
The criterion score for each module is also given. The body of the report 
is a student-by-module score (percentage) for all students in the section 
over modules taken up to the time of the report. Students are identified 
by both name and student number. The percentage scores shown for each module 
are scores after ret^st if a rttest was administered. An **R** is appended to 
the test score if a retest has been administered. 

A module sumnary report is specific to instructor and section as 
above but is also specific to a particular module. The report is headed 
by instructor name, section number, module number and module criterion score. 
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followed by a listing of the total number of test items for each objective 
and the number of correct answers required to meet criterion for each ob- 
jective. The body of the report is a student-by-objective score (number 
correct) for all students over all objectives in the module. Students are 
identified by both name and student number. Raw scores are shown for both 
initial posttest and retest, if a retest was administered* Retest scores 
are shown only for those objectives taken on the retest (those on which the 
student did not reach criterion). Test administration dates (day, month) 
are shown for both test and retest. 

All disk reads/writes, both in the instructional program and the 
data pack program "set-up" and data summary operations are performed through 
an assembly language function, DISKI0, originally written by Thomas McMurchie 
at Florida State University. It was modified by The University of Texas 
Computer-Assisted Instruction Laboratory staff to provide the capability 
of treating counters and buffers as arrays for transfer and thus reducing 
the number of reads/writes required to transfer information to and from disk. 



FORMATIVE EVALUATION 



Student Sample 

The first broad scale use of the modules occurred during the 
Fall, 1972, semester. It was initially assumed that approximately 200 
students would be run, this number being considered adequate to provide a 
data base for evaluation and revision. The course coordinator and instruc- 
tors, however, requested that the modules be made available to all ST:udents 
registered in the course. Consequently, the modules were used by approxi- 
mately 560 secondary education students enrolled in the Educational Psychology 
332S course. Since these students are secondary education trainees, entering 
the formal education program during their junior year, their lower-division 
courses rc,/resent a variety of different disciplines. 

The students were envolled in 15 different sections of 3325 taught 
by 14 instructors. Five of the 15 sections were already involved in an ex- 
perimental program entitled "Proctor-Assisted Instruction," These five sec- 
tions differed from the ten regular sections in terms of classroom instruc- 
tional techniques and class schedule. As a result of differing requirements 
of these two groups, they were treated separately for purposes of scheduling 
and evaluation. 
Procedures 

The 330 students in the 10 conventional classes received the in- 
struction during September and early October, 1972. These students studied 
only four of the five modules since their instructors did not wish to include 
the Statistics module in the curriculum. The evaluation of the Computers in 
Education, Classroom Management, Tests and Measurement, and Cultural Differences 
modules reported here was based on these students. 

The 200 students and 30 proctors in the five proctor-assisted instruc- 
tion classes studied all five modules. These students raceived the instruction 
during October, early November, and the first week of December, 1972. The only 
data from these^students to be discussed in this report concern the evaluation 
of the Statistics module and pretest data for all modules. Data from the 
proctors have not been used for evaluation purposes and will not be reported. 
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Pre-Instruct1ona1 Evaluation . While it was desirable that the test 
item pool be evaluated through administration to students prior to their receiving 
instruction, a conventional pretest-instruction-posttest procedure was not con- 
sidered to be feasible. Only two test forms were available for each module and 
it was considered preferable to reserve both forms for post-instructional 
testing and retesting. The additional demands which conventional pretesting would 
place on both student and computer terminal time wero also considered to be un- 
desirable. The decision was made, therefore, to administer a paper and pencil 
pretest battery to a sample of students drawn from the 332S classes. The desire 
to maintain a degree of test security raised additional problems for pretest 
administration. While the simplest procedure would have been to administer com- 
plete tests from the different modules to individual students, this would have 
given each student tested complete exposure to one form of one module test. The 
approach adopted involved a matrix sampling procedure in which each student was 
administered a test consisting of items drawn from all tests. 

The total pool of 298 items was divided into six subtests of 49 or 50 
items each. Each subtest contained items drawn from both forms of all module 
tests and^all subtests contained approximately equivalent numbers of items from 
each of the module tests. An individual test item appeared on only a single sub- 
test and no subtest contained parallel items from the alternative test forms for 
a single module. 

The pretest was conducted with 100 students drawn from three of the five 
proctor-assisted instruction classes during the month of September, prior to their 
introduction to the modules. Testing was conducted in groups and was completed 
within a five-day period. Each student tested was randomly assigned to one 
of the six subtests. Fifteen to seventeen students completed each subtest, 
thus providing fifteen to seventeen responses to each of the 298 items in the 
total test item pool . 

Instruction and Posttest . Details of ttie procedures to be employed 
during the formative evaluation were determined in cooperation with the class 
instructors. Each student was required to attain a certain level of mastery on 
each module rather than on each objective within a module. Specifically, students 
were required to achieve a score of 75% on each module. If a student failed to 
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reach this criterion on his initial posttest, he was administered a retest 
based on the alternative test form. For all modules other than Statistics, 
the retest items coverad only those objectives on which the student scores 
less than 75% on his initial posttest. His final module score, following the 
retest, was a composite of his scores on objectives passed on the initial 
test and his retest scores on objectives failed on the initial posttest and 
retested. Since the Statistics module subtests were quite short, the decision 
was made to retest over all objectives in a subtest following a student's 
failure on that subtest. If a student failed to achieve criterion following 
a retest, .'.e was referred to his instructor for an oral exarr^ination. 

Students in the ten conventional classes were int ^oduced to the 
modules during the first regularly-scheduled class meeting. The mariual con- 
taining the instructional modules ( Educational Psychology Modules , 1972) and 
the materials from Educational Testing Service for the Tests and Measurement 
module were distributed (at no charge) and the students were given a brief 
lecture on Computer-Managed Instruction in general and the procedures to be 
followed in this particular instance. An outline of these procedures wis also 
provided in the manual's introduction to the modules. Each instructor informed 
his class of the criterion established for mastery, of the dates by which each 
module test was to be completed, and that the content of the modules would not 
be discussed in class prior to the completion date. 

Approximately one week was allowed for the completion of each module. 
An extra three days was provided for the completion of the first module. Computers 
in Education, to allow the students more time to becon^e familiar with the 
procedures. The order in which subsequent modules were assigned was as follows: 
Classroom Manacement, Tests and Management, and Cultural Differences. Essen- 
tially the same procedures were followed for the five proctor-assisted instruc- 
tion classes, with the exceptions that they were introduced to the modules 
one month later, and modules were assigned in a slightly different order: 
Computers in Education, Statistics, Tests and Measurement, Classroom Management, 
and Cultural Differences. 

Students were assigned to one of the two test forms on the basis of 
the student numbers by which they were identified by the computer program. 
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Students with numbers ending in an odd digit were assigned to test form A 
for their initial posttest on each module while students with numbers ending 
in an even digit were assigned to form B for their initial posttest. Students 
were assigned to one of the two different item sequences within each test 
form on a random basis. 

The students' first contact with the computer terminals was for 
purposes of a demonstratic .. This demonstration, which was treated as a 
componsnt of the Computers in Education module, consisted of a short computer- 
assisted instruction tutorial on the use of the computer terminals and a brief 
demonstration of a segment of a computer-assisted instruction program on 
English punctuation and grammarv To facilitate scheduling, and since no data 
were being collected, students were encouraged to sign up for demonstration 
appointments two at a time. 

The eight cathode ray tube (CRT) and typewri ter -keyboard terminals 
in the Laboratory terminal room were available seventy hours per week; noon 
until 10:00 p.m. on Mondays, 9:00 a.m. until 10:00 p.m. on Tuesday through 
Friday, and 9:00 a.,ra:luntil 5:00 p.m. on Saturday. Terminal time for the 
initial demonstration was scheduled in one-hour blocks. Terminal time for 
tests and retests was scheduled in one-half-liour blocks. 

Having received his instructional materials and instruction on the 
use of the terminal, a student began his study of the first module. When he 
thought that he could achieve the stated objectives for the module, he tele- 
phoned the computer terminal room to schedule a 30-minute block of terminal 
time. During the initial few tests, a student was signed onto the terminal 
by a proctor. During later tests, students signed themselves on. The first 
message displayed following sign-on was the students' name and a message 
which asked him to call the proctor if it was not his name which was displayed. 
This was to assure that students signing themselves on had entered their correct 
student number. 

After signing onto the terminal but prior to beginning a test, the 
student was administered an on-line version of a nine-item curiosity scale 
(Leherissey, 1972) concerning his interest in the reading materials. The 
module test items were then presented one at a time and the student responded 
by typing a 1 , a 2, a 3, or a 4. Immediately following his completion o*.* 



21 



the test, the student was administered a five-item state anxiety scale 
(Spiel berger , Gorsuch, & Lushene, 1970) asking him about his emotional 
state during the test. The student was then given feedback on the CRT 
display regarding his performance. The feedback display included the criterion 
percentage established by his instructor, his own percentage score over the 
complete module, and his percentage scores on the individual objectives. 
Objectives for which the student achieved or surpassed the criterion percen- 
tage were starred. The student was then asked whether or not he wanted a 
typewriter printout of the diagnostic report. If the student achieved criterion, 
he was finished with that module. If he failed, he was instructed to restudy 
the materials and to make an appointment for a retest. On a random bdsis, one 
quarter of the students were asked to complete a thirteen-iteni paper and pencil 
attitude questionnaire following each of the four modules. 

In general the same procedures were followed for retests with the 
exception that the curiosity scale was not administered, the student was 
tested on only those objectives failed on the initial test, and no attitude 
scales were administered. 
Evaluation Results 

Pretest Results . Data resulting from the pre-instructional 
evaluation are shown in Table 1. Recall that since no one student was ad- 
ministered more than one-sixth of the items from any one module test, the 
total test scores shown represent composites of the performance of all students 
in the sample. Scores are presented in terms of percentages in order to 
facilitate comparisons among tests and across objectives. The five subtests 
from the Statistics module have been presented separately. 

In general, the obtained scores were higher than had been anticipated, 
much higher, of course, than would have been expected strictly on the basis of 
chance (25%). To some extent, the scores were probably inflated (relative to 
subsequent posttest scores) due to the use of a paper and pencil test as 
opposed to testing at the computer terminal. Even when this potential error 
term is taken into consideration, it must be concluded that the tests contained 
some poorly constructed items to which the answers were obvious or that some 
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stude:its already possessed skills to l^e taught in the instructional modules. 
Undoubtedly both of these factors were present. Despite the fact that the 
obtained scores were somewhat nigher than had been anticipated, the results 
did indicate that, in general, the ^Performance of this sample of students 
was well below the desired level of competency. 

An item analysis of the data was conducted which provided the 
probability of a correct response for each item, the item's point bi-serial 
correlation with total test score and the distribution of responses across 
the four alternatives. Separate response distributions were also provided 
for high and low scoring students based on a median split of total subtest 
scores. While these detailed data will not be reported, they were used to 
provide one basis for revision of the module tests. 

Posttest Results . The percentages of students in the ten conven- 
tional classes who achieved the criterion of 75% correct on each of four 
modules are shown in Table 2. As may be noted, there was some attrition (19%) 
from the first to the last mocule. For the most part, this was due to stu- 
dents' dropping the course d'lring the first month of the semester, although 
there were a few students who did not complete the last posttest prior tc the 
date on which data collection for this report was terminated. 

The percentage of students achieving criterion on the initial post- 
tests was lower than would eventually be desired, particularly for the fourth 
module, Cultural Differences, In general, however, U was considered to be 
satisfactory in view of the modules' stage of development. Student perfor- 
mance on retests was at least as good as had been anticipated and as a result 
the percentage of students eventually achieving criterion, on either the 
initial test c the retest, was quite satisfactory for three of the four 
modules . 

The Cultural Differences module, while definitely in need of revi- 
sion, probably suffered to some extent from its placement at the end of the 
sequence of modules. Since the final deadline for completing the modules was 
approaching, many students probably allocated less time to studying this module 
than they had to the others. Unfortunately, the Cultural Differences module 
contained the largest amount of reading materials of any of the modules. The 
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effects of the deadline are aiso indicated by the fact that wmle ]2b stu- 
dents failed to achieve criterion on the initial posttest on Cultural 
Differences » only 79 of these students scheduled a retest prior to the final 
data collection date. This discrepancy partially accounts for tne relatively 
low percentdg** of students finally achieving criterion, 78;c. It should be 
noted that Cultural Differences was the one module which had not been pilot 
tested anu revised during the summer session. 

The percentages of students in the five proctor-assisted instruction 
classes who aoieved the criterion of 75% correct on each of the five subtests 
in the Statistics module are shown in Table 3. 

The percentage*^ of students achieving criterion on th? initial post- 
tests are comi arable to the percentages for the other four modjles shown in 
Table 2. The percentages of students achieving criterion on r'^test are some- 
what lower thin the comparable percentages shown in Table 2. This discrepancy 
rnay be due to the nature of the subject matter or to the fact that students 
\,.iking Statis:ics were retested over all objectives in a unit rather than 
over only the particular objectives failed on the initial posttest. This 
iX)ssibility will be explored in subsequent research. The final percentages 
of students e/entually achieving criterion, on either the initial posttest or 
the retest, ii*re considered satisfactory in view of the stage of tne module's 
development. 

Thj mean percentage of items answered correctly on each of the 
initial posttest is shown in Table 4. Data are shown for both individual 
objectives, for the total test for each module, and for each unit subtest for 
the Statistic.! module. The magnitude of pre- to post-instructional gain may 
be Inferred by comparing these data to the data presented in TaMe 1, but it 
should be remenbered that the data in the two tables do not represent the 
same set of stidents. 

Equality of Forms and Seque nce. It will be recalled that two alter- 
native test forms were developed for each module. Alternative forms were 
assigned to students on a pseudo- random basis; that is, on the basis o^ the 
last digit of the.r student number. With the exception of the Statistics 
module. Items within each test form were also presented in two different orders. 
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Only a single sequence of items was developed for the Statistics tests due to 
disk storage limitations. While the alternative forms and sequences were 
designed to be of equal difficulty, an empirical check on this assumed equality 
was obviously necessary. For each of the first four modules, the total scores 
obtained by students assigned to each of the two item sequences and each of 
the two forms were subjected to. a two-by-two analysis of variance. Scores 
obtained by students assigned to each of the two forms of the Statistics post- 
test were subjected to a one-way analysis of variance. 

No significant differences were found between item sequences for 
any module. No differences were found between test forms for the Computers in 
Education or Classroom Management modules. Within the Tests and Measurement 
module, form B was found to have a lower mean score (22.4) than form A (24.5). 
This difference was significant (p < .001). Similarly, form B was found to 
have a significantly lower mean score (17.0) than form A (18.2) within the 
Cultural Differences module (p < .01). Differences in difficulty were found 
between forms "in two of the five-unit subtests in the Statistics module- 
Units I and II. In Unit I, form A was found to have a significantly lower 
mean score than form B (4.1 as opposed to 5.1) (p < .01). Form A was also 
found to have a significantly lower mean score than form B in the Unit II 
subtest (4.2 as opposed to 5.0) (p < .001). These differences were taken into 
consideration during revision of the modules in that test items for which per- 
formance data were available were distributed across forms so as to balance 
difficulty of individual objectives. 

Scheduling Considerations . A variety of time factors were also 
considered in the evaluation of the total CMI system. Specifically, the 
time required for students to complete their study of the modules, the terminal 
time required for students to complete the module posttests, and the pattern 
of scheduling for the posttests were determined. These data were based on 
only the 330 students in the ten conventional sections of the course and on 
the four modules studied by these students. 

It will be recalled that one of the constraints placed on the design 
of the instructional modules was chat, on the average, the time required for a 
student to complete his study of a module not exceed four hours. Since the 
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instructional system did not allow a means of directly monitoring the amount 
of time students spent on the different modules, one item on the module 
questionnaire asked the student to estimate the amount of time which he 
devoted to studying the module. The specific question was stated as follows 

How much time did you need to read all of the instructional 
materials for this module? Do not i nclude the time for the 
CAI demonstration in the first module. 

a. 1 hour or less . 

b. 1 to 2 hours. 

c. 2 to 3 hours. 

d. 3 to 4 hours. 

e. 4 hours or more. 

The questionnaires were administered to approximately one-quarter of the 
students fcrrowing their completion of each module posttest. A summary of 
their reported study times is shown in Table 5. 



Table 5 

DISTRIBUTION OF REPORTED STUDY TIMES 
TO COMPLETE EACH MODULE 



Module 


N 


Percentage of Students Reporting 
Each Amount of Time 






< 2 hours 


2-3 hours 


3-4 hours 


> 4 hours 


Computers in Education 


63 


30% 


35% 


25% 


10% 


Classroom Management 


70 


13 


26 


29 


32 


Tests and Measurement 


74 


9 


26 


28 


37 


Cultural Differences 


56 


2 


11 


25 


62 



30 



The Computers In Education module ms obviously not too long: 90% 
of the students completed the module in four hours or less. The Classroom 
Management and Tests and Measurement modules were marginal in that approxi- 
mately one-third of the students reported that they required more than four 
hours of study. The Cultur.-l Differences module was definitely too long, 
with only one-third of the students reporting that they completed their study 
of the module in four hours or less. 

A second consideration involved the amount of terminal time required 
for students to complete the various module posttests. This information was 
available from the computer systein records. Since the lengths of th? retests 
were variable and never exceeded the lengths of the corresponding initial 
posttests, only initial posttest times were considered. The distributions 
of student terminal times required to complete th« four modules are sunmrized 
in Table 6. The terminal times reported include the c*me required for the 
student to complete the nine-item curiosity scale and the five-item anxiety 
scale as well as the test itself. No systematic attempt was made to isolate 
the time required for students to complete these scales but It is estimated 
that they required a total of four to seven minutes of the students' time. 



Table 6 

MODULE POSTTEST TERMINAL TIMES: PERCENTAGE OF STUDENTS 
COMPLETING TEST WITHIN EACH TIME PERIOD 



Module 


No. of 
Questions 


N 


Time in Minutes 






< 15 


15-20 


20-25 


25-30 


> 30 


Computers in 
Education 


15 


332 


89X 


8X 


2% 


IS 


OX 


Classroom 
Management 


20 


315 


48 


29 


15 


4 


4 


Tests and 
Measurement 


30 


302 


13 


29 


27 


20 


11 


Cultural 
Oi f frrences 


24 


270 


8 


24 


30 


19 


19 



31 



Student tenninal time was, of course, related to some extent to the 
number of quc^stlons on the test. It will be recalled that terminal time for 
the tests and retests was scheduled In 30-minute blocks. The shortest two 
tests. Computers In Education and Classroom Management, were completed by 
almost all students well within the scheduled block of time. The small per- 
centage of students who required more than 30 minutes to complete these tests 
was easil.v absorbed in the slack in the scheduling procedures. Larger but 
still tolerable percentages of students required more than 30 minutes to com- 
plete the Tests ano Measurement and Cultural Differences posttests. Both of 
these tests were shortened somewhat during revision. 

It Is Interesting to note that t^ere was a tendency for students to 
require more time to complete the Cultural Differences posttest than to om- 
plete the Tests and Measurement posttest, despite the fact that the latter con- 
tained 25% more test items. It will be recalled that students registered the 
poorest posttest performance on the Cultural Differences module. The higher 
relative difficulty of this test was reflected in the students* longer response 
latencies to the test items and thus in their increased terminal time. 

In view of the fact that a large proportion of a CMI system's 
operational costs are directly attributable to student terminal time, it would 
be desirable to reduce the amount of time required for posttefts. To maintain 
a workable schedule, tne next shorter time period which would be considered 
would be 20 as opposed to 30 minutes. Such a schedule would ^llow three test- 
ing sessions per hour rather than the current two, a one-third reduction in 
the required terminal time. Assuming that the curiosity and anxiety scal^^ 
were deleted from the testing session or at least substantially shortened, it 
would appear that the maximum test length allowable would be approximately 20 
Items . 

As was previously mentioned, students were given deadlines by which 
they were to have completed the posttest (and retest. If necessary) for each 
module. Initially, nine days were allowed for the Computers in Education 
module and six days were allowed for the completion of each of the three 
successive modules. Prior to the expiration of the second deadline, both the 
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instructors and students requested that it be extended. Consequently, all 
deadlines were extended slightly. Seven days were allowed for the completion 
of the second and third modules. The rinal deadline was constrained by the 
fact that the second group of students had begun their testing sequence. 
Consequently, this completion date was allowed to slip only one day, allowing 
just four days for the completion of the Cultural Differences module. It 
should be remembered that these dates were final deadlines. Students were free 
to take the posttest for a particular module prior to the completion date for 
the previous module. 

The cumulative percentages oi" students who had taken each module 
posttest (and retest, if necessary) by a particular date are shown in Table 7. 
The figures shown in this table are percentages of the number of students who 
were administered each test and retest prfor to the final data collection date. 
As may be seen from these data, there was a tendency on the part of the stu- 
dents to procrastinate. In general, a large number of students waited until 
the last or next to last day before the deadline to take the posttest. On the 
average, however, only about 10% of the students failed to schedule posttests 
prior to the assigned completion dates. Retest scheduling presented a more 
serious problem. Since a student could not know whether a retest would be 
required until he had taken his initial posttest, and since most posttests 
were scheduled relatively late in the assigned period, only about 60% of the 
retests were scheduled prior to the deadlines. 

The most noticeable result of students* tendency to delay taking 
the posttests was the lack of regularity in terminal scheduling. The number 
of test sessions scheduled per day (both initial posttests and retests) over 
a one month period is shown in Figure 3. As is obvious from inspection of 
this figure, terminal usage was definitely not evenly distributed over time. 
Although terminal use never reacfied the maximum daily capacity of the system 
(208 test sessions), the four days on which the number of student tests exceeded 
100 did raise problems of finding suitable appointment times for all students. 
Figure 3 represents a total of 1566 tests and retests. A steady level of about 
75 tests per day would have permitted all of the students represented to have 
completed their testing by the final deadline without interfering with other 
activities using the computer terminals. 
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The assignment of completion dates for individual modules would 
appear to be necessary for the context in which these modules were used. 
Given the pressures of other class assignments, it is not surprising that 
the students tended to procrastinate. In the future, however, the assigned 
deadlines for different classes will be staggered in an attempt to alleviate 
the very uneven terminal demands experienced. 



REPORT OF ESTIMATED COSTS 

Deveiopmental Cos ts 

The estimated direct costs o1 developing the data management 
system and the rive computer-utinayed instructional modules are reported in 
Table 8. Indirect costs, such as fringe benefits and institutional over- 
head, have not been included since these would vary from one institution to 
another. Not all of the costs reported in Table 8 were incurred by the fund- 
ing agency, some being absorbed by the institution and the individuals involved. 
These figures do not include the cost o*^ revision following the formative 
evaluation. 

Since personnel time constituted the major expense of instructional 
development (approximately 83%), a special procedure was employed in an attempt 
to insure the accurate r3Cording of the expenditure of time. Each individual 
was given a form and asked to maintain a weekly record of the amount of time 
spent on the project. Several of the project personnel failed to maintain 
accurate records and the time expended by these individuals was estimated. 
l...anever possible, the accurately recorded time expenditures for the perfor- 
mance of similar tasks was used as a basis for these estimates. 
Operational Costs 

The operational costs associated with the use of four rriodules 
(excluding Statistics) for the 330 students in the ten conventional classes 
are reported in Table 9. Since procedures had been established for recording 
the costs of supplies and computer time prior to the implementation of this 
project, no speci^dl procedures were required to insure the accurate recording 
of these operational costs. Thus, t^^ information in Table 9 was obtained 
through a simple analysis of tne rec ds normally maintained by the Computer- 
Assisted Instruction Laboratory. 
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Table 8 

ESTIMATED DEVELOPMENT COSTS FOR THE DATA MANAGEMENT SYSTEM 

AND THE FIVE INSTRUCTIONAL MODULES 

Estimated Time Estimated 
in -HouFS Total Hours Cost 

Data Management Systems Development 
Personnel 

Management 35 

Systems Analyst 121_ 156 805.00* 

Computer (IBM 1500 System) 

Author time @ $12.18/hr 25.9 

Utility time @ $48.73/hr 27.9 412.00 

1 ,217.00 

Instructional System Development . 
Personne l 

Management 415 
Authors and Instructional Designers 1246 
Systems Analyst 311 
Computer Programmers 415 
Keypunch Operators 35 
Secretarial /Clerical 380 

Instructors _87 2889 9,602.00* 

Computer 

CDC 6600 System @ $260/hr 0.2 
IBM 1500 System: 
Author time & $12.18/hr 44.3 

Utility time @ $48. 73/hr 63.7 1 ,527.00 

Copyright Fees 210.00 

Supplies and Materials 100.00 

11 ,439.00 

♦Estimated salaries excluding fringe benefits 
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Table 9 
CMI OPERATIONAL COSTS 
(First 300 students using four modules) 



Personnel 



Time in Hours Total Hours Cost 



Management 
Secretarial /Clerical 
Proctors 
Operators 



35 
69 
363 
190 



657 



$ 1 ,740.00* 



Computer (IBM 1500 System) 

Student time @ $6.C9/hr 
Utility time @ $48.73/hr 



822.0 

16.5 838.5 5,810.00 



Instructional Materials 



377.00 



$7,927.00 



*Salaries excluding fringe benefits. 



REVISION OF THE MODULES AND TESTS 



Following collection of the formative evaluation data, the five 
modules all underwent substantial revision. This revision was based on three 
sources of data: (1) student perfomance from the pre-instructional evalua- 
tion; (2) student performance on the initial posttests; and (3) student 
responses to the attitude questionnaires administered following each module. 
In contrast to norm-references testing situations, no reliable procedures 
have been generally accepted for evaluating criterion-referenced tests or 
for diagnosing the problems underlying poor performance on these tests. Con- 
sequently the procedures followed in the revision process were more heuristic 
than algori thmic . 

As a first step, the data from the pre-instructional evaluation 
were examined. If the mean probability of a correct response was found to be 
particularly high (greater than .60), that item was examined closely for con- 
struction deficiencies. This examination was guided by the pattern of responses 
to the distractor alternatives. Specifically, the following four questions were 
asked regarding the item: (1) Are the distractors plausible? (2) Are the 
distractors grammatically consistent with the stem? (3) Are there verbal 
associations between the stem and the alternatives? (4) Are the relative 
lengths of the alternatives equivalent? If no serious deficiencies were rioted 
in any of the items assigned to a particular olgective and if the menn 
probability of correct responses of items assigned to that objective was 
relatively high, the question was raised as to v'hether that objective should 
be included in the instruction. 

A much larger csta base was, of course, available for the post- 
instructional evaluation. In addition to the performance data described in 
this report, all of the students' responses to individual test items were 
recorded on magnetic tape. A card containing responses made to each item 
on his^ initial posttest was punched for each student for each module. All cards 
for each test form within each module were then subm'itted to an item analysis 
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program (Jennings, 1970) run on the University's CDC 6600 computer. This 
program divided the students into pantiles based on their total test score 
and indicated the number of students within each pantile selecting each 
al terr^ati ve. The item^s overall probability of being answered correctly 
and its point bi-serial correlation with the total test score were also 
i ndi cated. 

If the mean probability of a correct resp-'^nse to a posttest item was 
found to be particularly low (less than .75), the item was examined closely for 
construction deficiencies. Specifically, the following four questions were 
asked regarding the item: (1) Is the stem meaningful? (2) Does the stem 
present a definitv^ problem? (3) Does the stem contain irrelevant material? 
(4) Is the correct answer the only correct response or clearly the best 
response? In this Cdsey even more reliance was placed on the pattern of 
responses to the distractor alternatives. If the item contained no obvious 
deficiencies, the instruction itself was suspect. The relative response pat- 
terns of students falling into the different pantiles were then examined. If 
the item tended to discriminate students in the lowest pantile (the bottom 20% 
with respect to total score) from students in the upper four pantiles, the in- 
structional problem was not considered to be too serious. If, on the other 
hand, the pattern of iii'i.orrect rasponding was relatively even across all levels 
of overall student performance, a clear case was made for altering the instruc- 
tion. Modification of the instruction was also considered if the mean correct 
response probability for all items pertaining to a particular objective was 
relatively low. 

The third source of information for revision consisted of student 
statements recorded on the attitude questionnaires. If a relatively large 
percentage of students indicated that they required more than four hours to 
complete ■:heir study of the module, the amo^jnt of reading material assigned was 
reduced. This was done without eliminating objectives whenever possible. Stu- 
dents were also requested to indicate which of the readings in each module they 
would prefer to have eliminated. Alternative n-^adings were sought to replace 
those readings which a particularly large proporvion of students nominated for 
elimination. 
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Only minor changes were made in the instructional materials for 
the Computers in Education module. Several paragraphs were rewritten for 
clarity, one of the readings was replaced with a mere interesting paper, 
and the on-line demonstration of computer-assisted instruction was deleted. 
Approximately one-quarter of the test items were replaced or modified. 

Similarly, the revision of the Classroom Management module was 
relatively minor. Some definitions in the supplementary materials were 
rewritten for clarity and approximately one-fifth of the test items were 
revi sed. 

Portions of several readings in the Tests and Measurement module 
were deleted, the series of readings was reorganized, and one paper was added 
for the purpose of providing an organizational overview. In all, the length 
of the module was reduced by one-quarter. In conjunction with this reorganiza- 
tion, several of the objectives were rewritten and the number of objectives 
reduced from 12 to 'iO. The number of test items on each form was reduced from 
30 to 20. Of these 20, approximately one-quarter were either new items or 
items which had been modified. 

The Cultural Differences module presented the major revision prob- 
lem. It will be recalled that the module was much too time-consuming and that 
student performance on this module was substantially below that of the other 
four modules. The total length of the module was shortened by one-half. This 
was accomplished by deleting one reading and replacing two others by brief 
papers written by the module authors. Four of the 12 objectives were deleted 
and two others rewritten to reflect these changes. The number of test items 
per '"orm was correspondingly reduced from 24 to 15. Of these 15 items, 
approximately two-thirds were either revised or new items. 

It will be recalled that the Statistics module was divided into five 
units with a separ^ite posttest for each unit. This was found to be a relatively 
unsatisfactory situation. Too much computer time was required to administer 
the tests and students objected to the requirement that they make repeated 
trips to the computer terminal room. When faced with a long series of tests, 
students who failed an initial posttest tended to retake the test immediately 
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rather than spend any substantial amount of time studying and returning to 
take the retest at a later date. Consequently, the module was reorganized 
into two rather than five units. Based on student performance, stuJent 
comments on the attitude questionnaire and feedback from the course instruc- 
tors, the material previously contained in one of the five units was deleted 
from the moduie. The number of objectives was correspondingly reduced from 
16 to 13. A short supplementary reading was added to clarify a concept not 
adequately covered in the original materials The total number of test items 
for the module was reduced from 60 to 37 per form of which approximately one- 
third were new or modified. 

The introduction to the set of modules was revised and expanded to 
provide students with more complete information regarding the procedures to 
be followed in the CM! portion of the curriculum. Finally, a five-page sec- 
tion was added to the manual which provided the students with a brief tutorial 
on the nature of criterion-referenced tests and the appropriate use of objec- 
tives as a guide to studying. 

The revision of the five modules and their tests was completed by 
January 15, 1973. They are currently being used by approximately 450 student 
registered in the course during the spring semester. A second evaluation of 
student performance and attitudes will be conducted following thei r. completion 
of the modules. It is anticipated that the revisions described above will 
result in an increase in postt3st performance levels. 
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APPENDIX A 
COMPUTERS IN EDUCATION 



COMPiaERS IN EDUCATION 

Clair* W^:-:n*5t^ :r. 
Department of Educational Psychology 

The University of Texas at Austin 
Sunc^r, 1972 

Instructional Materials: 

I. Introduction to the i*lodule 
II. Demonstration of a Computer- Assisted Instruction Program 
(Sutton Hall 314) 

11a. Reading Assignir.ents with Objectives and Sample T<*st Questions 
IV. Six Mimeographed Readings 

I. IHTRODUCTION TO TilL MODULE 

This !T!Odule is designed to provide the student with an introduction 
to one aspect of the radical ch^inges taking place in educational thought 
and practice. As our conceptions of the school and its place in society 
change, so do the roles of tiie student and the teacher. The w^e of con- 
puters in the schools represents one of the areas of inncvation destined 
to have a major impact on our total educational environment. 

The materials you will read have been written or selected to introduce 
you to the principles underlyinj^ the design of programmed instructional 
materials (of which computer-assisted instruction is one version) and to 
prt)vide you Cith an overview of a num£>er of ways in which computers may 
be used in an educational setting. Two of the papers in the module discuss 
changes in the role of the teacher and the activities of students which can 
be expected when computer technology is introduced into a classroom. The 
final pai^r in the module is a brief assessment of the current positive 
aspects and shortcomings of instructional computer applications. 

This module is not intended to sell the concept of computer applica- 
tions to education. Hopefully, computers will be adopted for particular 
instructional applications on the merits of their usefulness in those 
specific sit»iations. it is quito probaole, however, that you will en- 
counter some instructional computer application in your teaching career. 
This mxlule i£ intended to provide you with enough information, negative 
as well as positive, to allow you to decide hov you might use the computer*s 
capabilities (if at all) in your ovm classroom. 

As you work your way through this series of modules* you will be 
participating in one form of aii instructional computer application-- 
cofi^uter- managed instruction. Unfortunately, you» as a student, will not 
see much of how the system operates. Your only exposure to the computer 
will be wliile taking tests or obtaining prescriptive assignments. To give 
you a better understanding of the computer's instructional capabilities 
a demonstration program has been prepared as one component of ti.is module. 
This program will teach you to use the computer terminal and includes a 
short segment of a nighly interactive computer-assisted instruction pro- 
gram. Make an appointment to see this demonstration as soo n as possibl e. 
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The complete appoi'.tfneut will take about one hour. We suggest that you 
get together with a friend and make a single appointment for the two of 
you. We have found tnat such demonstrations are much more interesting 
if two or more people work through them together. 

It should take you about thj?ee hours to complete the readings in this 
module. Study the article description, obiective{s ) , and sample test 
item(s) before reading each paper. The purpose of the objectives is to 
help you direct your study of tne materials, l^en you have completed the 
module, you will be giver, a 15-item muitiple-clioice test. If you review 
the sample test items given with the objectives, they should give you a 
good indication of whether or not you are ready to take the mastery test, 

II. DEMONSTR/TION CF A COMPUTER-ASSISTED INSTRUCTION PROGRAM 
(Sutton Fill 31**) 

III. READING ASSIGNMENTS, OboiXTIVLS^ AND SAMPLE TEST ITEi-IS 

After completing each of the reading assignments, you should have 
achieved the objective(s) given for that assignir^ent . A sample test item 
is prov:^ (led oeicw eac. objective to allow you to assess your mastery of 
tnat oDjeci"v«. V^.e correct ar*:^wer is starred (*). The inastery test 
you will ta<e :it ti;e computer termiaai upon completing these readings is 
composed of items similar to the sample items. 

Reading 1 : Computer- Assisted I nstr ucri on (CAI) by Wilson A. Judd 

This reading serves as an introduction ,o the use of computers in 
education by discussing what many consider to be their most interesting 
educational applicat ion--computer-assisted instruction or CAI. CAI is a 
fonn of progranmed instruction. To provide a context in which to view 
CAI, instructional prograiWiiing is discussed as a process for the develop- 
ment of instructional materials. Following a orief description of two 
earlier forma of programrned instruction, the general characteristics of 
CAI ^re discussed. Three specific types of CAI programs are then described: 
drill and practice; tutorial; and dialogue. Some of the assuiripticns under* 
lying these types of programs are examined and a fairly recent innovation, 
xeamer- controlled CAI, is described. 

ubjective 1 

The student will dent if y characteristics of instructional materials 
design for pr-^grainned instruction and CAI. 

Sample Test Item 

Which of the following is nc>t a characteristic of instructional 
materials developed for CAI? 

The materials are auto-instructional. 
Tlie materials ^re individualized. 

All revisions are completed beforr the materials are 
administered. 

The materials provide the student with feedback on his 
[>erf omance . 
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Objective 2 

The student will identify the definition of CAI and characteristics 
and examples of the following three types of CAI programs; (1) drill and 
practice; (2) tutorial; and (3) dialogue. 

Sample Test Item 

A high school Spanish class uses the school's computer terminals to 
practice their vocabulary and spelling skills. They are most probably 
using a 

(1) simulation program, 

(2) tutorial program, 

'•(3) drill and practice program. 
(^) dialogue program. 

O bjective 3 

The student will identify characteristics and examples of learner- 
controlled (as opposed to conventional) computers-assisted instruction. 

Sample Test Item 

A drill and practice program in arithmetic is designed so that 
problems of a given type continue to be presented to the student until 
he indicates that he is ready for the next problem type. This is an 
exampl3 of 

(1) computer-managed instruction 
*{2) learner-controlled instruct: 

(3) simulation, 
computer-based evaluation. 

Reading 2 : The Use of Computers for Instructional Simulation 



This reading consists of two parts; a general description of the 
ways in which computers can be used to simulate real-life situations for 
instructional purposes and an article (The Application of Computer Tech- 
nology to the Improvement of Learning by Don D. Bushnell) which discusses 
several specific instructional applications of computer simulation. Par- 
ticular emphasis is placed on the teaching of disadvantaged children. 

Objecti ve ^ 

The student will identify the reasons for computer simulation and 
the benefits that simulation offers for disadvantaged children. 

Sample Test Item 

Which of the following is not a circumstance in which computer 
simulations are appropriate substitutes for real-life experiments? 

(1) When the time scale causes difficulty. 

(2) When danger is involved. 

(3) When equipment is unavailable because of expenses or complexity. 
liJhen a qualified teacher is not available. 
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Sample Test Item 

Mrs. Jones' school has just installed a computer-managed instructional 
system. As a teacher, her rcle will now be that of a 

(1) lecturer. 

(2) clerk. 

(3) disseminator of information. 
''^i^) manager of the learning process. 

Reading 6 : General Assessment of C om^; ut er-Assisted Inst r uctional Systems 
by Wilson A» Judd - 

This final reading is presented for the purpose of attempting to place 
the current and future educat^.onal roles of computers in their proper per- 
spective. The computer is potentially a very powerful instructional medium. 
Cvirrently, there are relatively few applications in which the computer's 
use has begun to approach this potential. Many technical, pedagogical 
and social problems remain to be solved. There will be no test questions 
on this reading. 
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APPENDIX B 
CLASSROOM MANAGEMENT 



W. PaLil Scott 
Department of Ediicationai Psychology 

and 

Donald u. Rogers 
Department of Curriculum and Instruction 

The University of Texas at Austin 
iumner , 1972 

Instructional Materials . 

I. Introduction to the Module 
II. Cognitive Oojectives and Sample Test Questions 
III. Supplenientary !;aterials 
IV. Mink, 0. G. , The Beha vior Change Process . New York: 
Harper 6 Row , 'i970V " Paperback , S**.95. Available at 
tne Co-op and Hemphill's under your instructor's name. 

I. INTRODUCTION TO THE MODULE 

This module is designed to give ycu a new vocabulary to use in 
discussing classroom discipline problems and some insights about the 
way teachers and students influence each other's behavior. You will 
be introduced to some specific techniques for changing undesirable 
student behavior and replacing it with more appropriate behavior. 

The materials you will read are a relatively short paperback by 
Oscar Mink, The t^havior LLange Proc ess, and some supplementary 
materials prepared here at the University of Texas. The book begins 
with two short pretests. You may find it interesting to take thesvi 
tests to find out how mucii you already know ai^out the material. Tiie 
text itself is divided into two parts. A brit.t introductory section 
introduc€S you to the way a 'behavioral" psychologist looks at student- 
teacher interactions and discipline problems and presents the basic 
termlnolog[y you'll De learning. Tne major segincnt of the book is a 
programmed course on the various techniques of changing student 
behavior. Since the material is presented in one unit, rather than 
in topical chapters, you will probably find it helpful to read the 
prograraned portion of the book in as few sittings as possible. Our 
experience thus far indicates that 851 of the students complete tlink's 
book in four hours or less* 

To assist you in your study, we have prepared a short set of 
supplementary materials vhich define and discuss some terms which we 
consider to be particularly critical to your unoerstaniing of this 
topic. The definitions of the terms have been indented, so that you 
can find them quickly. 



After you've read the material^ yoa will oe f,iven a 20-itt'r. nultiple- 
choice mastery test. As is true for all the modules you'll use this semester* 
the items on the mastery test wore written to measure your mastery of a set 
of cognitive oojectives. The cogTiitive objectives for this modu-^e are listed 
on the following pages. Each is a statement of a specific learnia^ F,oal for 
these materials. We have included a sample test item corresponding to each 
object ive . 

We suggest that you begin by studying the cognitive objectives and 
corresponding sample test items. Next, read over the supplcmentarv n.^terials 
to acquaint yo\irself with tlie new terminology. Then study the textbook, 
finally » study the supplementary materials again to be sure that you aider - 
stand the terms defined there. It is a good idea to use the postrests at 
the end of the textbook aijd to review the sample test items f^iven L^elow to 
determine whether or not you are ready to take the mastery test, 

II. COGNITIVE OBJECTIVES MD SATIPLE TEST QUESTIONS 

After completing this module, you will have achieved the following 
objectives* Notice that below each objective we have included a sample 
test question designed to assess your mastery of that objective. The correct 
answer is starred (^»). The mastery test you take upon completinr; your study 
of Ilink's programmed text will be con\posed of items similar to these. 

Objective 1 

When given the name of any term or process defined in the supplementary 
materials, the student will select the correct paraphrased definition of 
that tenn or process. 

Sample T e st Item 1 

A positive reinforcer 

(a) is tlie addition of ar. avers ive stimulus. 

(b) decreases the frec^uency of undesircible behavior. 
•'(c) increases the frequency of tiie behavior it follows. 

(d) is the opposite of a punisher. 

Objective 2 

When given a description of any of the processes defined in trie 
supplementary materials, the student will correctly identify ana label 
that process. 

Sample Test Item 2 

Applying an aversive stimulus is an example of 

(a) extinction. 

*'*(b) punishment. 

(c) shaping. 

(d) negative reinforcement. 
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Objective 3 

When given a list of descriptive phrases, the student will correctly 
Select those phrases which descrii^e observable behavior. 

S ample Test Item 3 

Which of the following is a behavior? 

(a) thinking 

{ b ) under s t and i :\g 

(c) knowing 

•♦(d) touching 

Objective U 

iVhen given a description of a student-teacher interaction in a specific 

classroom situation or a general history of student-teacher interactions, 

the student will select the most likely future teacher and student behaviors. 

Sample Test Item **a 

If, by asking silly quest ions » a student continually g^ts the teacher's 
attention » the student will probably 

(a) stop asking questions. 

*''(b) continue to ask silly questions, 

(c) try to Ask an intelligent question, 

(d) find other ways to get attention too. 

Sample Test Item ub 

The teaciier decided to ignore e very silly question. Host students will 
probably 

(a) stop asking silly questions immediately, 

(b) ask more intelligent questions, 

(c) continue to ask silly questions* 

-'(d) begin to ask silly questions less frequently. 

Objec t vve 5 

When given a description of a specific class-teacher or child-teacher 
interaction, the student will correctly identify and label the components 
of that interaction. 

Sample Test Item 5 

In the situation above (r^a^iple test itefns Ua aikd *ib), the teacher 
learned that her attention 

(a) negatively reinforced the desired response • 

(b) negatively reinforced the undesired response. 

(c) positively reinforced the desired response. 
*(d) poaiti/ely reinforced the undesired response. 
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Objective 6 

When given a description of the class-teacher or child-teacher inter- 
action, the student will correctly select the raost appropriate teacher 
responses for producing a desirable change in student behavior. 

Sairple Test Item 6 

In the situation above (sample test itens '♦a and Hb), in order to 
increase appropriate questions the teacher should 

(a) continue to extinguish silly questions by ignoring tnem. 

(b) punish silly questions every time they occur. 

(c) counter condition new verbal responses, 

**(d) shape more relevant questions by continually attending to them. 
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TESTS A-;D KEASUREMEiO' IK IKE CiJVSSROOf^ 

Paul I;ixon and Claire Weinsteln 
Jepartnent of Educational Psychology 

The University of Texas at Austin 
Surorocr, 1972 

Instructional Materials: 

I. Introduction to the :jodule 
II. Reading Assignments with Objectives and Sample Test Questions 
III. Five Micieographed Readings Included vith the Module 
IV. Two booklets from the Educational Testing Service (ETS) Tests 
and Measurement Kit: (i) Selecting an A chievement Test , and 
(2 ) M aking the Classro om Test 

I- INTRODUCTION TO THE MODULE 

This module is designed to familiarize you wit); a variety of evaluation 
procedures and instruments for use in the classroom. The characteristics and 
educational ur^^s of standardized aptitude and achievement testr* are discussed. 
A number of technical labels and terms are presented to facilitate your under- 
standing and use of such tests < The various purposes of testing are discussed 
and» finally, a set of general and specific guidelines for developing your ovm 
classroom tests are presented. 

The materials you will read consist of five short papers (included with 
the module) and two booklets from the Educational Testing Service (ETS) Tests 
and Measurement Kit. ETS is a noiiprofit organization, located in Princeton, 
New jersey, devoted to measurement and research in educatioii. You will un- 
doubtedly come Into frequent contact with ETS products and services throup,hc/ut 
your teaching career. There is a wide variety of other materials included in 
the kit which will prouaLly oe of interest to you now and in your future career. 
ETS^s various testing programs and services are described together with infor- 
mation about several specific tests. Two booklets, one on preparin?, multiple- 
choice questions and another on short -cut statistics for classroom tests, should 
be of particular interest. Only two of the booklets, however. Selecting an 
Achj.evemen t Test and leaking the Classroom Test, have been assigned as readings 
for tMs module. 

Of a previous group of students studying a sli^^htiy longer version of 
tliis module, 81% indicated that they completed the readings in four hours or 
less. After you have read the nw sriai, you will be given a 30-iten rrult i plr»- 
cfioice mastery test. If you review the sample test items given with the obj^.c- 
tives after you have studied the iraterials, they should give you a goou indi- 
cation of whether or not you are ready to take the mastery test. 
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II. ^lEADIIvG ASSIGNI:EIJTS, objectives, and sample test OUESTIOiJS 

After completing each of the reading assignments, you should have 
achieved the objectives given for eac:: assignment. VJe have provided a sample 
test question below eac:h objective designed to assess your mastery on that 
objective. Tne correct answer is starred (-'0. The mastery test you v;ill take 
upon completing the readino;s will be composed of items similar to these. 

Reading 1 : Gath_erijig ^^Information About Students by C. A. Cartwright 
(adapted for Ciil format). 

Tlie purpose of this reading is to provide tlie student with an introductory 
overview of some basic ideas about data collection in the classroom. 

Objective 1 

The student will recognize the characteristics of quantitative and 
qualitative information by identifying examples of each and by recognizing 
the rationale for i^eporting observed behavior. 

Sample Test Item 

Qualitative information is 

"(i) i^ot as precise as quantitative information. 

(2) numerical information. 

(3) obtained through objective testing. 

(^•) used only when we can precisely measure a specific ability. 
Objectij^e_2 

The student will recognize the various steps in the teaching ■ learning 
process by identifying examples of and the characteristics of each step in 
the process. 

Sample Test Item 

Vv'hich of the following is not a step in the teaching-leai^ning process? 

(1) Provide feedback to the learner. 

(2) Select and use teaching procedures. 

(3) Formulate objectives. 
Assign student graaes. 

Reading 2: Evaluation Procedures by C. A. Cartwright (adapted for C'/il format). 

Tl'iis article provides a urief overview of the types of standardized tests 
used in the classrooi^i. 

Objec tive _3 

The student will recognize tL^e cliaracteristics of aptitude and achieve- 
ment tests and identify examples of eacii. 
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Sample Test Item 

Aptitude tests are 

''•(1) designed to measure a student's capacity or potential. 

(2) designed to ir.easure accomplishment in a subject area. 

(3) seldom standai-dized. 

(4) usually administered individually by the teacher. 

Reading 3 : Reli ability, Validit y, and Usability by D. Schreiber and 
C. a/ Ca!rtvn?ight (adapteT for^CMI format). 

This reading describes various considerations which are important in 
selecting an existing testing instrument or the improvement of teacher-made 
instruments. 

Objective 4 

Given examples of the types of reliability and validity, the student 
will recognize characteristics of each, and will identify the relationship 
between reliability and validity. 

Sample Test Item 

A test that yields similar scores upon repeated administration to the 
same individuals is said to have high 

(1) validity. 
"•(2) reliability. 
(3) causality 

none of these 

Reading 4 : Selectin g an Achie vement Tes t : Princip l es and Procedures , 
Educational Testing"*^ervTceT (The reader may exclude the 
sections on validity and reliability.^ pp. 7-11.) 

This booklet provides the student with a number of guidelines for the 
proper selection of standardized tests for classroom use. The sections on 
reliability and validity in this article will not be tested. They are, however, 
recoininended as an aid to understanding the material in Reading 3. Moreover, 
no objectives require computational procedures. 

Objec tive 5 

Given examples of groups to be tested, the student will select the 
proper norm group. 

Objective G 



The student will recognize the various considerations necessary for the 
selection of the proper achievement test. 
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Objective 7 

The student will recognize the purposes of testing and identify the 
characteristics of each purpose. 

Sample Test Item 

IThich of the following is not a use of tests? 



(1) placeTfient analysis, 

(2) diagnosis analysis. 

(3) assessment analysis. 
-(4) subjective analysis. 

Reading 5 : I nterpret in g Tes t Scores Rea listica lly, author unknown. 

This treading offers a number of important considerations necessary for 
correct interpretation of test scores from both standardized and teacher-made 
tests. A test score is only an estimate of the student's ability at any 
given time and it takes on meaning only when it is properly interpreted. 

Objective 8 

The student will interpret the relationship between two pupils' test 
scores xvhen given (a) the two scores and (b) the standard error of measurement 
for the test. 

Objective 9 

The student will recognize the 'guiding principles' of test interpretation. 
Sample Test Item 

When interpreting tests, the teacher must always remember that 

(1) a student's score alone indicates the grade he should receive. 

(2) a student -s score indicates his true ability. 

(3) "true score' is another term for the scores obtained on any given 
test. 

"(4) percentiles gain meaning only when the characteristics of the 
norm group are known. 

Readin g 6 : Tea cher Devised Achievement Tests_ by C. A. Cartwright (adapted 
for CMI "format") . 

This section offers a general overview of the various methods used in 
the construction of teacher-made tests. Several question formats are dis- 
cussed. 

Objective^ 1£ 

The student will recognize the specific characteristics and special uses 
of each of the question formats discussed. 
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Sample Test Item 

If we wished to measure recall of information that has been stored in 
memory, we would use 

'^Kl) completion items. 

(2) multiple-choice items. 

(3) alternative response items. 
(U) matching exercises. 

Reading 7 : Making the Classroom Test: A Guide for Teachers , Educational 
Testing Service. 

This booklet is a practical guide to the construction of better test 
questions. Don't lose it! It will be a very helpful aid when yov are actually 
confronted with your first test construction task. 

Objective 11 

When given an example test question, the student will recognize three 
obvious pitfalls that should be avoided in writing test questions: specific 
determiners; extreme reading difficulty; and ambiguity. 

Objective 12 

The student will recognize the limitations and advantages of both essay 
and objective test questions. 

Sample Test Item 

An objective test item 

(1) cannot tap high levels of reasoning. 

(2) covers a narrow field of knowledge. 

(3) can be scored acci:.rately and consistently, 
all of the above. 
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CULTURi\L DIFFERENCES 

Brenda Rutherford and Charles D, Saddler 

Department of Educational Psychology 
The Unive-^sity of Texas at Austin 
Fall, 1972 



Instructional Materials : 

I, Introduction to the Module 
II. Reading Assignments with Objectives and Sample Test Questions 
III. Six Readings 



I. INTRODUCTION TO THE MODULE 

This module is designed to familiarize you with certain aspects of 
cultural differences between the Anglo middle-class culture and the 
blcL^k and Mexican -American cultures. The major areas of cultural diffe 
ences covered in this module are language and standardized test perfor*- 
mance differences and how these differences affect school achievement. 

The materials you will read consist of six articles drawn from a 
variety of sources* The first two articles deal with differences in 
language jDetween the black and Mexican-American cultures and the Anglo 
middle-class culture. The following two articles discuss IQ and stan- 
dardized test performance differences on the part of blacks and Mexican 
Americans. The fifth article summarizes cultural differences in lan- 
guage, IQ, and test performance, and concludes with a discussion of how 
these differences are relevant to the present educational situation. 
The final article presentn a formal discussion of cultural differences 
in intellect and how these differences affect school performance* , 

It is anticipated that you should complete the readings in four hour 
or less . After you have read the material , you will be given a multipl 
cho.ice mastery test. If you review the sample test items given with 
the objectives after you have studied the materials, they should give 
you a good indication of whether or not you are ready to take the mas- 
tery test. 



II. READING ASSIGNMENTS, OBJECTIVES, AND SAMPLE TEST QUESTIONS 

After completing each of the reading assignments, you should have 
achieved the objectives given for each assignment. The sample test 
items provided below each objective are designed to assess your mastery 
of that objective. The correct answer is starred The mastery 

test you will take upon completing the readings will be composed of 
items similar to these. 
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Reading 1 : Academic Ignorance and Black Intelligence: by William Labov. 

Labov discusses the basic tenets of the verbal deprivation theory 
aiid then presents evidence to refute the theory . He demonstrates that, 
although black children appear to be non-verbal in certain situations, 
they are quite verbal in a familiar environment wV\en using the Black 
English Vernacular ( BEV ) ♦ Labov presents an interesting discussion of the 
merits of middle and lower class soeech, questioning the dominance of 
middle class values in education. He concludes that the failure cf edu- 
cation has been a result, not of the verbal d^-anrivation of the black 
children, but of the inability of the school to adjust and adspt to 
Cultural differences. 

Objective 1 

The student will identify characteristics of the verbal deprivation 
theory as it relates to Black English. 

Sample Test Item: 

According to the verbal deprivation theory, black children in 
the urban ghetto areas 

1. participate fully in a highly verbal culture. 

2. seldom participate in verbal interchanges with peers. 
*3. cannot foriTi concepts or convey logical thoughts. 

4. have the same basic vocabulary as white children. 

Objective 2 

The student will identify characteristi.es of Labov' s linguistic 
refutation of th^ verbal deprivation theory and recognize the 
implications for teaching. 

Sample Test Item: 

According to Labov, one serious consequence of the verbal 
deprivation theory is its negative effect on 

•^1. the teacher's attitude toward the student. 

2. interpersonal relationships among students. 

3. :he attitude of the student toward the teacher. 

4. the types of inter /iewo which are successful. 



Reading 2 : Bilingually Advantaged , by Judith Rae Gates . 

Gates presents four viewpoints prevalent in approaches to educating 
Mexican-American children. She mentions the bilingual approach and gives 
justification as to why it is better than any of the other approaches. 
The remainder of the article discusses a bilingual program currently 
being impleiriented in San Antonio, Texas. 
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Objective 3 

The student will identify differences between the English as a Second 
Language (ESL) approach and the bilingual approach for the education 
of Mexican -American children. 

Sample Test Item: 

The bilingual approach for the education of Mexican-American 
children differs from the English as a Second Language (ESL) 
approach in that 

1^. the bilingual approach is based on methods used in teaching 
a foreign language. 
*2. the ESL approach is based on methods used in teaching a 
foreign language. 

3. the ESL approach requires that 400 English words be learned 
at the beginning. 

4. the bilingual approach initially makes no use of English 
for classroom interaction. 

Objective 4 

The student will recognize argunients for the bilingual approach as 
presented by the author and as demonstrated in the San Antonio 
system. 

Sample Test Item; 

The bilingual approach as implemented in the San Antonio school 
system is advantageous because it 

1. teaches the student 400 words in English in the first grade. 

2. groups students according to their knowledge of English. 
*3. makes the curriculum more relevant to Mexican-American 

experiences . 

4. tests the students to evaluate their knowledge of English. 



Reading 3 : The I.Q., Chapter 2 in Herbert Ginsburg's The Myth of the 
Deprived Child . 

Ginsburg discusses four major ''myths'' concerning intellif^ence (IQ) 

tests. He illustrates why these myths are not valid. Ginsburg also 
discusses the controversy concerning the effects of environment and 
heredity on I .Q. 

Objective 5 



The student will recognize the shortcomings of measured "intelligence" 
(defined by IQ tests) as described by Ginsburg. 
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Sample Test Item: 

According to Ginsburg, an important quality which IQ tests do 
not measure is 

1. verbal skills. 

^*2» creativity. 

3 . memory . 

^ . perception . 

Objective 6 

The student will identify characteristics of heredity and environment 
that may affect performance on IQ tests. 

Sample Test Item: 

Ginsburg presents evidence to show that one of the most impor- 
tant factors affecting IQ scores is 

1 . age . 

2. sex. 

3. number of siblings. 
'^K, socio-economic status. 



Reading M- : Assessing Assessment Instruments: A Chicano Perspe^.tiva , by 
Ernest M. Bemal, Jr. 

BeTO.al discusses the testing situation as it currently applies to 
Mexican-Americans. He lists six ways in which the use of tests may dis- 
criminate against Mexican-Americans. However, he believes that testing 
can serve a legitimate purpose if used for understanding and planning, 
rather than merely labeling. He concludes by recommending 10 courses of 
action to be taken in order to end the discriminating use of tests with 
Mexican-Americans. (It is important to realize that, although Bernal 
confines his discussion to test discrimination against Mexican-Americans, 
what he says can be applied to many minority groups.) 

Objective 7 

The student will identify ways in which tests are in'^Doropriately 
used with members of minority groups. 

Sample Test Item: 

Which of the following is a way in which test«^ eve inappropriately 
used with members of minority group.*:? 

1. Using t(jsts fo"A? program modification. 

2. Using tests for diagnostic purposeii . 

*3. Using tests that are irrelevant to minority group experience. 
^. Using tests that are criterion-referenced. 
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Objective 8 

The student will \dentify (a) reasons why tests designed for Anglos 
should not simply be renormnd and used with Mexi can -Ameri cans , and 
(b) Bemal's recommendations for improving the testing of Mexican- 
American students. 



Sample Test Item: 



Which of the followinf: does Bernal recommend in order to improve 
the testing of Mexican-American children? 

1. Test Mexican -American children only after lunchtime. 

2. Use only Mexican-American testers to test Mexican-American 
children. 

3. Use only tests which are written in ^'standard'* Spanish. 

4. Develop new testing instruments and procedures to minimize 
the gap between ethnic groups. 



Reading 5 : Cultural Differences and Inferences about Psychological 
Processes, by Michael Cole and Jerome S. Bruner. 



Cole and Bruner discuss two interpretations of ethnic and social 
class differences: the deficit interpretation, which postulates that poor 
children are deficient because of their environment; and the difference 
interpretation , which postulates that poor children are not deficient, but 
riifferent in that they demonstrate skills which are different from tradi- 
\::ional middle-class skills. Evidence is presented which supports the 
latter interpretation. Cole and Bruner relate this evidence to the con- 
cepts of competence and performance. The remainder of the article is 
devoted to a discussion of the relationship between these two concepts, 
and the implications for teaching. 

Objective 9 

The student will identify characteristics of the concepts of compe- 
tence and performance, especially as they relate to "cultural 
deprivation. " 



Sample Test Item: 



Which of the following is true regarding the concepts of compe- 
tence and performance discussed by Cole and B '^uner? 



1. Culturally deprived people have no competence. 

•2. Competence is inf'^tTed from performance. 

3. Performance is i.f;fer??ed from competence. 

4. Per-^ormance is a hypothetical coristruct. 



Objective 3.0 



The student will identify implications from Cole and Bnaner's 
presentation for teachers cf ''disadvantaged'* children. 
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Sample Test Item: 

An implication of the arguments set forth by Cole and Bruner is 
that the teacher of "disadvantaged" children should 

1, tolerate classroom disorder, 

2, use only traditional materials in the classroom, 

3, maintain order at all costs, 
use relevant study materials. 



Reading 6 : Ir.tellect and the Schools, Chapter 5 in Herbert Ginsburg's 
Th e Myth of the Deprived Child , 

Ginsburg lists the assumptions underlying the government subsidized 
compensatory education programs currently in operation and examines the 
fallacies in each assumption. He then makes a strong case for new and 
innovative practices to be employed in the education of poor children. 

Objective 11 

The student will recognize the assumptions underlying compensatory 
education and Ginsburg's refutation of these assumptions. 

S^ample Test Item: 

Compensatory education was created under the assumption that it 
would be 

1. a method of making education more equitable. 

2. the answer to the present educational crisis in urban schools. 
'-^S. a means of enhancing school perfonrance in poor children, 

^, a met:.od of providing day care services for poor working 
mothers . 

Objective- 12 

The student will identify the characteristics of traditional education 
which, in G;.nsburg's opin.'on, must be changed to make education 
beneficial to poor children. 

Sample Test It^^m: 

According to "Sinsburg, traditional educational practices include 

*1, a relatively small cjnount of fi aedom for the student, 
2. open class;^oom situations for the student, 
1, use of individualized instrcction, 
purple water fountairs. 
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STATISTICS IF THE CLASSROOM* 

Paul W, Dixon 
Department of Educational Psychology 
The University of ""exas at Austin 

Fall, 1972 

Instructional Materials ; 

I. Introduction to the Module. 
II. Reading assignments with objectives and sample test questions. 
III. Hereford, C. F. , Natalicio, L, F. S., and lIcFarland, S. J., 

Statistics and Measu r ement in the Classroom . Kendall/Hunt , 1969 . 
Paperback, $3.75. Available at the Co-op under your instructor's 
name . 

IV. ''Statistics; Tools for Better Teaching," a mimeographed paper 
by Paul N. Dixon, included with the nodule. 
V. A Glossary of statistical terms and symbols, included with the 
module . 

INTRODUCTION TO THE MODULE 

The purpose of this module is to provide you with a number of statistical 
methods useful in the organization and interpretation of data from teachers- 
made tests and standardized tests. 

The materials you will study consist of selected sections from a book of 
readings. Statistics and lleasurement in the C la ssroom , a mimeographed paper 
included with the module, rind a Glossary of statistical teinns and symbols. 
This module differs from the others in this program in that it is cjivided into 
f ive units . Each unit has its own set of readixig assignments and objectives. 
Also, rather than having one long test at the end of the module, there is a 
short multiple-choice teet for each of the five units. The lengths of the 
tests are as follows: Unit 1-6 items. Unit II - 6 items. Unit III - 16 
items Unit IV 20 items, and Unit V - 12 items. The testing procedure 
for this module differs from that of the other modules in that if you fail to 
reach criterion on the first test, you will be retested with an alternate, 
form of the complete test rather t!\au being retested over just the individual 
objectives failed on the first test. Since some of the tests are quite 
short, you might wisn to take more than one test during a single session. 
You will be able to continue to a second test only ±£^ you reach criterion 
on the first test. If not, you vill have to make another appointment to take 
the retest before continuing to the next unit test. 

Of the crc)up of 25 students who have previously studied this module, 
71% indicated that they completed the readings in four hours or less. 

RliADOG ASSIG/:^MENTS, CBJECTIVES, AID SA>-lPLE TEST QUESTIONS 

After completing each of the reading assignments, you should have 
achieved the olijertives given for that assignment. A 8a;aple test question 

*U8ed by permission of the author^ copyright pending,. 
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is provided for each of the units designed to assess your mastery of tnat 
unit. The correct answer is starred (*) • The mastery test you will take 
after completing each unit is composed of items similar to these. 

UNIT I 

USES OF STATISTICS AND INTRODUCTION TO lEKAS 

Reading 1 ; "Statistics: Tools for Better Teaching,'' by Paul Dixon 
and page ix, Statistics and Measurement in the Classroo m. 

These readings explain why the teacher should study statistics and 
they describe in general terms the use of statistics in the classroom. 

Objective 1 

The student will identify the basic uses of statistics in the classroom. 
Objective 2 

The student will recognize the characteristics of evaluation discussed 
in the readings j; the kinds of evaluation and their respective roles in the 
classroom. 

Re::J iug 2i "How to Talk Back to a Statistic/' by Hull and Gels, Statistics 
and Measu r emen t In the Cla ssroom , pp. 59-74. 

This article offers yet another reason why the teacher and most anyone 
else in our society should have some understanding of statistics. \Jith the 
constant use of statistics in our news media ^ a basic understanding helps 
the teacher sort out distortions (through the almighty number) of many issues. 

Objective 3 

Given an experimental situation (e.g., sample selection), the student 
will recognl:5e the validity or lack of validity of the statistical procedure. 

Ic eading 3 ; ''Statistical Terms and Statements,^' by Allen Edwards, Statistics 
and Ilea surement in the Classroo m, pp. 3-7. 

This article provides expanded definitions of various statistical 
terme found in your glossary and later in the readings. The article should 
be used in conjunction with the glossary as you may wish to refer to it when 
further explanation of the following terms is needed. 



1. 


variability 


2. 


relationship 


3. 


average 


4. 


sample 


5. 


prediction 


6. 


population 


7. 


statistic 


8. 


parameter 


9. 


statistical inference 


10. 


confidence limits 


11. 


confidence coefficient 


12. 


tests of significance 



Ther^ will be no tt^stlr.g over t.ils article. 
Sapp le Tes t I ten for in Is Unit 

Statistics le to assecsi^ent evaluation as 

1. testing Is to JuugTDfcnlal evaluation. 

2. ecores are to subjective analysis. 

3 . asscssGient evaluation Is to j u Jginen t al evaluation, 
subjective analysis is to judj^ental evaluation. 

irNIT II 

VARIABLLS ANU SCALES 

P eading 1 : "Variables and .scales,' by Ailci. i,uu'ards Statistl^cs anji 
M easurc g>ent in t ne C lassrooru, pp. 9-21. 

Thi» reading i« designed to provide tne reader wltb a definition ol 
the term "variable' and a description of the various laeasureaent scales 
and their application. 

Object ive 1 

The student will identify examples of (a) or<;an ismic . behavlorjl, 
stimulus and response- inferred organisiaic variar^les, (b) the four scaltb 
of measurement, Cc) discete and continuous variables. 

O bjectiva 2 

Thd* student will recognize ^liaracteristics of (aj tht four scales of 
measurement, and (b) discrete ana continuous variables. 

Sample Tes t Item for this Unit 

The cieasurenent scait: in whicii the zero point is fixed is the 

1. nominal scale. 

2. ordinal Gcale. 

3. interval scale. 
**4. ratio scale. 

UNIT III 

FREQUEPJCY DISTRIBbTIONS , :u.ASlJRx:S OF CENTRAL 
TENDUJCV AI^D TliE NORMAL CURVE 

Reading 1 : 'Staulaticai Analysis of l^aca," by Arnold Lien, Statis tics and 

iia aaur<rment in the C las sroom, pp. 23-30 (read to part F, p. 30), 
nnJ p. 41, sunaary statements 1-5. 

This section acquaints the reader witn the methods of gruupin;; data 
(buch as teat scores) and determining the three measures cf central tendency, 
mean, meulan, arid mode. 
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Given a set of raw scores, the student will construct a HimpK' 1 rcquenv 
distribution and compuie the mea.; , median, and Qode for th it distribution ot 
scores . 

Objective 2 

Glvr-n 3 specific sltujtion, tlie student will c^-.oose the aeab^re of 
centra] tendency best useo in those circumstances, recognize the chai^i^.terl^^ 
tics of the three measures of central t endency* -naean , median, and mode. 

R eading 2 : "The Concept of the No rcial Curve, by Susai^ McF.irland, ^tatiytlc 
cinr\ lea^ureme nt in t_he L las ^ r oom , pp. A 3-4 7. 

This article further explains irethods of nandllnp data in addition to i 
description of the theoretical normal curve. 

Objective 3 

Given a g^^P^ of scores, the student will identify and differentiate a 
frequency polygon and a histogram and recognize the characteristics of each. 

Obje ct ive A 

The student will recognize four cUirciC' er ist ics of th^i theoretical 
normal curve. 

Sample Test Item for this 

Which of the foilowln ; Is a characteristic of the mean, medlnn, and mod 

*1. rhey may be statletlcs or parameters. 

2. They are all oeasures of variability. 

3. They are not subject to sampling error. 
A. None of the above. 

UNIT IV 

I^SUllhS OF VARIABILITY 

;ve ac.l.i^ 1 "Statistical Analysis of Jata, by Arnola Lien, S^tat 1 st i cs and 
f^eas urem en t in f f^lassroom, pp. 30-36 (read part F only), and 
p . 41, summary s t atemen t 6 . 

This section describes three measures of variability and the uethods 
for calculating each. 

Objective i 

Given an incomplete calculations table, the student will complete the 
table (see table 9, p. 8A , Statistics and Measurement in the Cla ssroom ) . 



Objective 2 



The student will recognize the characteristics of the external range, 
range by quartiles, and standard deviation and given a set of data on a cal- 
culations table (see table 9, p, 84, Statistics and Measurement in the 
Classroom), and the necessary formulas, calculate each. 

Reading 2 ; "Uses of the Standard Deviation," by Abraham Franzblau, Statistics 
and Measurement in the Classroom , pp. 49-57. 

Having learned how to compute the standard deviation, this article 
explains why you go to the trouble I 

Objective 3 

The student will recognize the relationship of the standard deviation 
to the normal curve. 

.Objective 4 

Given a specific testing situation, the student will choose the best 
interpretation of the results and identify aspects of the standard deviation 
useful to the interpretation of test scores. 

Objective 5 . 

Given test scoreu and the standard deviation for two groups, the 
student will rank plac.-iment on these tests. 

Sample Test Item for thi s Unit 

If we wish to compare John's score on Test A with his score on Test B, 
which of the following statistical tools would be most helpful in providing 
a meaningful picture of his performance? 

1. Range and correlation coefficient for the two tests. 

2. Correlation coefficient and standard deviation for the 
two tests. 

3- The median, mode, and range for the two tests. 
*4. The mean and standard deviation for the two tests. 

UNIT V 

CORRELATION 

Reading 1 ; "Statistical Analysis of Data,^ by Arnold Lien, Statistics and 
M easurement in the Classroom , pp. 36-41 (part G only) and 
pp. 41-42,. summary statements 7-10. 

This section gives a basic description and definition of correlation 
and it explains computation for the p(rho) or rank-difference correlation 
method. 



Objective I 



Given an ur.comple ted Lorrclatlon table {see table 10, p 38, 
Stati stics and ."leaaureDe nt i.'i the Cla^s room) and the formula for a rank- 
difi"erence correlation, fhe student will complete the steps In calculating 
the table and conpute a rank-ui - i t fence correlation coefficient. 

The student will recoji^nize varyifig degrees of relat ionshi apd tiie 
Cv'o basic klu<ls of correlation trom givtn exaupies, identity unwarranted 
caasality stargments dssriv^d f ^om given data* and recognize tne \'ariou8 
ch^racterifi ties of correlation. 

Sair^^ le Te st Item for this UiUc^ 

The correlation coefficient 

1. indicates central tendency. 

2. indicates which of two factors causes the otl.er. 

3. indicates an inverse relationship when its value is zero. 

*A . can indicate as much relationship in a negative direction as In 
a positive direction. 
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Sector 16 



Word 0 



RESERVED 



14 : 



I ^ 

I 

15 ' Course Count (Max. = 1) 
I 

1^ 



16 i Section Deletion Indicator 



17 



18 



19 



20 



21 



.22 



23 



Pack Initialization Indicator 



Beginning Sector Addr. of Course 
Configuration Record (17) 



FWA ui Course Configuration Info 
(0) 



Sector Address of Instructor File 



Current Student Count (< 640) 



Position (1-640) of Head of Chain 
of Avail. Student Recording Blocks 



Current Section Count (< 60) 



Sector 1^ (continued) 



24 



Up to 60 2-character 
section identifiers 



83 



84 



85 



143 
144 



319 



Pointer (1-20) 
to secti on file 
with info on 
entry in word 24 



Position (1-3) 
of entry in 
word 24 within 
list of i nstruc- 
tor's sections 



Ur to 59 additional 
entries corresponding 
positionally to entries 
i n overal 1 1 1 i st of 
sections (^4-83) 



rese|^vld 



Figure 1. Structure of the recording pack directory (bookkeeping information) 
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Sector 17 



Word 1 \ Total number course modules (;^25) 

I 



^ Nur'her objectives for module 1 



Number questions for objective 1 



21 I Number questions for cbiective 20 





295 I .Nupbe; obiii-cives for modvile 15 
298 



315 
316 

319 



Number questions for objective 1 



Number questions far objective 20 



Reserved 



ERIC 



Sector 18 



21 





Reserved 


-1 


Number 


obiecti ves for 


— , ^ 

module 16 


Numhe r 


Questions for 


objecti ve 1 


r ^ 
1 

! 

■ 


Number 


oueotions for 


obiectivp 70 



190 



191 



210 
211 



Number objectives for module 25 



Number questions for obiective 1 



Number questions for objective 20 



31? 



Res 



r 'ed 



Figure 2, Structure of the course ccnf 1 guration record 
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l^'orcl 0 



37 



Logical Pack No 



S 
C 



3 


I 


0 

i 


4 


u 




5 


F 


I 


6 


L 


E 

1 


7 




» 1 


8 


Pointer to next entry in 

of avai 1 . section f i les 

1 


chain ; 


9 


Maximum section count (3) 


i 
1 


10 


Current section count \ 


1 1 


Current student count of 


instruc- I 



12 



36 



tor's 1st section (v 60) 



Module no. of Isti 

module to be ^ i Criterion (%) 

taken 



Module no. of 
25th module to 
be taken 



Criterion (%) 



88 

89 



148 
149 



268 
269 



319 



Up 10 2 additiondl 2b-word 
section cor.f iigura 1 1 on records 
for instructor's 2nd and 3rd 
secti ons 



60-/vurd toble pointers 
to recording : kjks for 
students ( 1 -60 , of f i rst 
secti on (entries range 
1-640) 



Up to 2 additional 60-word 
tables fur stjudents of 2nd 
and 3rd sections 



Figure 3. Structure of class section conf i qf^rati on information (disk sectors 22-Al ) 



ERIC 



F-4 



Module Record (8 words) 



Retest No. 

0 ^ No. 1 

1 ^ Nc. 2 



0 ->■ Form A 

1 -> Form B 



1 J 

1 1 

Module No. , Curir'-.ity* 
(< 25) 1 Scale No. 


' Anxi ety 
' Scale No 




0 4 1 5 


10 ; 11 


15 


1331 x 0^ +121 X 02 + n X 






1 
1 
1 

1 


^ 

1 
1 

1 


1 

i 
1 

1 


1331 X + 121 X + 11 






1 

, fetal module latency 


in seconds 




0 5,6 


10 |n 


15 


1 

Test Form ( p, 
Indicator i 

1 

L .. . 


' Month 

I 

1 





- M X 2" + C X + A 



^. is student's 
Fxore (total correct) 
on objective i 
(Max. = 10) 



^Note: 



Sectors 300-1579 are designated as recording blocks. Each 
recording block consists of 2 sectors. Each student registered 
is allocated one block. Witltin that block, a series of 8-word 
records is written, ona record for each module test/retest. 

Test/retest is indicated by the value of the curiosity scale number. 
Test is indicated by a positive value. Retest is indicated by zero, 



Figure 4» Structure of student data files (sectors 300-1579) 
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